Image processing method and image processing device

ABSTRACT

An image processing method and an image processing device are provided. The image processing method includes the following steps. A light source is projected onto a 3D model via a plurality of projecting locations to generate a plurality of facial shadow pictures. The facial shadow pictures are superimposed to generate a composite shadow picture. An eye-eyebrow location, a nose location and a mouth location in a 2D facial map are obtained by way of analyzing according to the composite shadow picture. A plurality of characteristic points are calculated according to the eye-eyebrow location, the nose location and the mouth location in the 2D facial map.

This application claims the benefit of Taiwan application Serial No.107131918, filed Sep. 11, 2018, the subject matter of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates in general to a processing method and a processingdevice, and more particularly to an image processing method and an imageprocessing device.

Description of the Related Art

With the evolution of image processing technology, a face synthesizingtechnology has been developed. The human face synthesizing technologycan synthesize one human face onto another face. The human facesynthesizing technology can be applied to products, such as virtualreality (VR), augmented reality (AR), mixed reality (MR), animation,movies and the like.

In the human face synthesizing technology, the most important issue isthe need of obtaining the representative facial feature characteristic.Conventionally, the user has to manually select the characteristicpoint. However, the artificial selection of the characteristic point hasthe too low accuracy, and the facial synthesis result tends to getdistorted. In addition, if the selection of the characteristic point istrained by way of machine learning, a lot of computation resources needto be spent. Thus, the selection of the characteristic point forms agreat bottleneck in the facial synthesis technology.

More particularly, when the human face synthesizing technology isapplied to the 3D facial model and the accuracy is too low, thedistortion condition becomes more serious. In addition, the computationresources for the human face synthesizing technology applied to the 3Dfacial model become huger. If the spending of the computation resourcescannot be effectively reduced, it is difficult to apply the human facesynthesizing technology to the 3D facial model.

SUMMARY OF THE INVENTION

The invention relates to an image processing method and an imageprocessing device that use different projecting locations to generatedifferent shadows, and thereby automatically calculate the facialcharacteristic points. In the human face synthesizing technology, theinvention can be used to improve the precision of selecting thecharacteristic points, and increase the image processing speed.

According to a first aspect of the invention, an image processing methodis provided. The image processing method includes the following steps. Alight source is projected onto a 3D model via a plurality of projectinglocations to generate a plurality of facial shadow pictures. The facialshadow pictures are superimposed to generate a composite shadow picture.An eye-eyebrow location, a nose location and a mouth location in a 2Dfacial map are obtained by way of analyzing according to the compositeshadow picture. A plurality of characteristic points are calculatedaccording to the eye-eyebrow location, the nose location and the mouthlocation in the 2D facial map.

According to a second aspect of the invention, an image processingdevice is provided. The image processing device includes a processor.The processor is used to perform an image processing method, and theimage processing method includes the following steps. A plurality offacial shadow pictures are superimposed to generate a composite shadowpicture. The facial shadow pictures are generated by projecting a lightsource onto a 3D model via a plurality of projecting locations. Aneye-eyebrow location, a nose location and a mouth location in a 2Dfacial map are obtained by way of analyzing according to the compositeshadow picture. A plurality of characteristic points are calculatedaccording to the eye-eyebrow location, the nose location and the mouthlocation in the 2D facial map.

The above and other aspects of the invention will become betterunderstood with regard to the following detailed description of thepreferred but non-limiting embodiments. The following description ismade with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart showing an image processing method according toone embodiment.

FIGS. 2 and 3 are schematic views showing a step S110.

FIG. 4 is a schematic view showing a step S120.

FIG. 5 is a schematic view showing a step S130.

FIG. 6 is a schematic view showing a step S140.

FIG. 7 is a schematic view showing a step S150.

FIG. 8 is a schematic view showing a step S160.

FIG. 9 is a schematic view showing steps S170 to S190.

DETAILED DESCRIPTION OF THE INVENTION

The technical features of the invention are described in detailhereinbelow with reference to the embodiment, which is not intended tolimit the scope of the invention. In the image processing method of thisembodiment utilizes a light source providing different projectinglocations to generate different shadows so that the facialcharacteristic points are automatically calculated. In the human facesynthesizing technology, the invention can be utilized to enhance theprecision of selecting the characteristic point and increase the imageprocessing speed.

FIG. 1 is a flow chart showing an image processing method according toone embodiment. Referring to FIG. 1, in this embodiment, the automaticcalculation of characteristic points is performed through steps S110 toS140 in order to facilitate subsequent various image processingprocedures, which are, for example, procedures for synthesizing picturesinto a 3D facial model in steps S150 to S190. An image processing methodin this embodiment can be performed by a processor 110 (see FIG. 2) ofan image processing device 100 (see FIG. 2). The image processing device100 is, for example, a computer, a server, a cluster computing system,an edge computing system, or a cloud computation center.

FIGS. 2 and 3 are schematic views showing the step S110. Referring toFIGS. 2 and 3, in the step S110, a light source 200 is projected onto a3D model 300 via a plurality of projecting locations A1 and A2 to enablea camera 400 to generate a plurality of facial shadow pictures P11 andP12.

In this embodiment, because the projecting locations A1 and A2 aredifferent from each other, shadows rendered by the facial shadowpictures P11 and P12 are different from each other. The projectinglocations A1 and A2 may be an upper front, a lower front, a front leftor a front right of the 3D model 300. The designer may designappropriate directions and locations to make the shadows rendered by thefacial features in the 3D model 300 more complete. For example, theprojecting location A1 in FIG. 2 is the upper front at an angle of 45degrees, and the projecting location A2 in FIG. 3 is the lower front atan angle of 15 degrees. In an embodiment, the number of the facialshadow pictures can be greater than two, and more facial shadow picturescan make the shadows rendered by the facial features more complete.

The 3D model 300 may be a 3D human facial model or a real person. Theshadow of the 3D model 300 may truly indicate the relative location andrange of the facial features, and the situation of the mis-judgementcannot occur.

FIG. 4 is a schematic view showing the step S120. Referring next to FIG.4, the processor 110 superimposes the facial shadow pictures P11 and P12to generate a composite shadow picture P2.

In the step S120, the processor 110 firstly performs shadow pointenhancement on the facial shadow pictures P11 and P12 to generateenhanced facial shadow pictures P11′ and P12′. For example, the shadowpoint enhancement is to perform the process of binarization, backgroundremoval or noise removal on the facial shadow pictures P11 and P12.Next, the processor 110 superimposes the enhanced facial shadow picturesP11′ and P12′ again to obtain the composite shadow picture P2.

It can be obtained from FIG. 4 that the facial shadow picture P11obtained by the single projecting location does not apparently renderthe complete facial features, but the composite shadow picture P2obtained by the multiple projecting locations can apparently render thecomplete facial features.

FIG. 5 is a schematic view showing the step S130. Referring to FIG. 5,in the step S130, the processor 110 obtains a contour boundary patternP3 of the composite shadow picture P2 according to the composite shadowpicture P2. In this step, the boundary search algorithm is, for example,used to analyze the contour boundary pattern P3. Then, the processor 110analyzes an eye-eyebrow location L1, a nose location L2 and a mouthlocation L3 in a 2D facial map P4 according to the contour boundarypattern P3. In this embodiment, the 2D facial map P4 corresponds to afront side angle of a 3D facial map P9 (shown in FIG. 9). In anembodiment, the angle corresponding to the 2D facial map P4 only needsto be consistent with the angle captured by the 3D model 300, and theaccurate eye-eyebrow location L1, nose location L2 and mouth location L3in the 2D facial map P4 can be analyzed smoothly.

FIG. 6 is a schematic view showing the step S140. Referring to FIG. 6,in the step S140, the processor 110 calculates a plurality ofcharacteristic points CP according to the eye-eyebrow location L1, thenose location L2 and the mouth location L3 in the 2D facial map P4. Inan embodiment, the processor 110 calculates the characteristic points CPby way of uniform distribution. For example, the number of thecharacteristic points CP corresponding to the eye-eyebrow location L1may be equal to 22, the number of the characteristic points CPcorresponding to the nose location L2 may be equal to 9, and the numberof the characteristic points CP corresponding to the mouth location L3may be equal to 19.

After the step S140 is finished, the plurality of characteristic pointsCP of the 2D facial map P4 are obtained, and various image processingprocedures including, for example, a procedure for synthesizing picturesinto a 3D facial model (i.e., steps S150 to S190) are performedaccording to these characteristic points CP.

FIG. 7 is a schematic view showing the step S150. Referring to FIG. 7,in the step S150, the processor 110 marks the characteristic points CPon a human face picture P5. In this step, the processor 110 can firstlymark the characteristic points CP containing a jowl on the human facepicture P5, and then filter out the characteristic points CPcorresponding to the eye-eyebrow location L1, the nose location L2 andthe mouth location L3.

FIG. 8 is a schematic view showing the step S160. Referring next to FIG.8, in the step S160, the processor 110 extracts a convex hull range R0from the human face picture P5 according to the characteristic pointsCP. The convex hull range R0 is formed by the characteristic points CPcorresponding to the eye-eyebrow location L1, the nose location L2 andthe mouth location L3. The convex hull range R0 covers several mostimportant organs of the human face.

FIG. 9 is a schematic view showing the steps S170 to S190. Referring toFIG. 9, in the step S170, the processor 110 maps the convex hull rangeR0 to the 2D facial map P4. That is, the image of the human face pictureP5 corresponding to the convex hull range R0 are mapped to the convexhull range R0 of the 2D facial map P4.

Next, in the step S180, the processor 110 performs a skin colorequalization procedure on the 2D facial map P4, so that the skin colorinside the convex hull range R0 approximates to the skin color outsidethe convex hull range R0.

Then, in the step S190, the processor 110 synthesizes the 2D facial mapP4 into the 3D facial map P9.

The human face picture P5 can be synthesized into the 3D facial map P9through the above-mentioned steps. Different facial shadow pictures P11and P12 are generated by using the different projecting locations A1 andA2 in the above-mentioned embodiment, so that the facial characteristicpoints CP are calculated automatically. The precision of selecting thecharacteristic points CP is improved, and the image processing speed isalso increased.

While the invention has been described by way of example and in terms ofthe preferred embodiments, it is to be understood that the invention isnot limited thereto. On the contrary, it is intended to cover variousmodifications and similar arrangements and procedures, and the scope ofthe appended claims therefore should be accorded the broadestinterpretation so as to encompass all such modifications and similararrangements and procedures.

What is claimed is:
 1. An image processing method, comprising:projecting a light source onto a 3D model via a plurality of projectinglocations to generate a plurality of facial shadow pictures;superimposing the facial shadow pictures to generate a composite shadowpicture; obtaining an eye-eyebrow location, a nose location and a mouthlocation in a 2D facial map by way of analyzing according to thecomposite shadow picture; and calculating a plurality of characteristicpoints according to the eye-eyebrow location, the nose location and themouth location in the 2D facial map.
 2. The image processing methodaccording to claim 1, wherein in the step of projecting the light sourceonto the 3D model via the projecting locations to generate the facialshadow pictures, the projecting locations are an upper front, a lowerfront, a front left or a front right of the 3D model.
 3. The imageprocessing method according to claim 1, wherein the projecting locationscomprise an upper front at an angle of 45 degrees and a lower front atan angle of 15 degrees.
 4. The image processing method according toclaim 1, wherein the step of superimposing the facial shadow pictures togenerate the composite shadow picture comprises: performing shadow pointenhancement on the facial shadow pictures; and superimposing theenhanced facial shadow pictures to obtain the composite shadow picture.5. The image processing method according to claim 1, wherein the step ofobtaining the eye-eyebrow location, the nose location and the mouthlocation in the 2D facial map by way of analyzing according to thecomposite shadow picture comprises: obtaining a contour boundary patternof the composite shadow picture; and analyzing the eye-eyebrow location,the nose location and the mouth location in the 2D facial map accordingto the contour boundary pattern.
 6. The image processing methodaccording to claim 1, wherein the step of calculating the characteristicpoints according to the eye-eyebrow location, the nose location and themouth location in the 2D facial map calculates the characteristic pointsby way of uniform distribution.
 7. The image processing method accordingto claim 1, wherein in the step of calculating the characteristic pointsaccording to the eye-eyebrow location, the nose location and the mouthlocation in the 2D facial map, the number of the characteristic pointscorresponding to the eye-eyebrow location is equal to 22, the number ofthe characteristic points corresponding to the nose location is equal to9, and the number of the characteristic points corresponding to themouth location is equal to
 19. 8. The image processing method accordingto claim 1, further comprising: marking the characteristic points on ahuman face picture; extracting a convex hull range according to thecharacteristic points; mapping the convex hull range to the 2D facialmap; performing a skin color equalization procedure on the 2D facialmap; and synthesizing the 2D facial map into a 3D facial map.
 9. Theimage processing method according to claim 1, wherein the 2D facial mapcorresponds to a front side angle of the 3D facial map.
 10. The imageprocessing method according to claim 1, wherein the 3D model is a 3Dhuman facial model or a real person.
 11. An image processing device,comprising a processor for performing an image processing method, theimage processing method comprising: superimposing a plurality of facialshadow pictures to generate a composite shadow picture, wherein thefacial shadow pictures are generating by projecting a light source ontoa 3D model via a plurality of projecting locations; obtaining aneye-eyebrow location, a nose location and a mouth location in a 2Dfacial map by way of analyzing according to the composite shadowpicture; and calculating a plurality of characteristic points accordingto the eye-eyebrow location, the nose location and the mouth location inthe 2D facial map.
 12. The image processing device according to claim11, wherein the projecting locations are an upper front, a lower front,a front left or a front right of the 3D model.
 13. The image processingdevice according to claim 11, wherein the projecting locations comprisean upper front at an angle of 45 degrees and a lower front at an angleof 15 degrees.
 14. The image processing device according to claim 11,wherein the step of superimposing the facial shadow pictures to generatethe composite shadow picture comprises: performing shadow pointenhancement on the facial shadow pictures; and superimposing theenhanced facial shadow pictures to obtain the composite shadow picture.15. The image processing device according to claim 11, wherein the stepof obtaining the eye-eyebrow location, the nose location and the mouthlocation in the 2D facial map by way of analyzing according to thecomposite shadow picture comprises: obtaining a contour boundary patternof the composite shadow picture; and analyzing the eye-eyebrow location,the nose location and the mouth location in the 2D facial map accordingto the contour boundary pattern.
 16. The image processing deviceaccording to claim 11, wherein the step of calculating thecharacteristic points according to the eye-eyebrow location, the noselocation and the mouth location in the 2D facial map calculates thecharacteristic points by way of uniform distribution.
 17. The imageprocessing device according to claim 11, wherein in the step ofcalculating the characteristic points according to the eye-eyebrowlocation, the nose location and the mouth location in the 2D facial map,the number of the characteristic points corresponding to the eye-eyebrowlocation is equal to 22, the number of the characteristic pointscorresponding to the nose location is equal to 9, and the number of thecharacteristic points corresponding to the mouth location is equal to19.
 18. The image processing device according to claim 11, furthercomprises: marking the characteristic points on a human face picture;extracting a convex hull range according to the characteristic points;mapping the convex hull range to the 2D facial map; performing a skincolor equalization procedure on the 2D facial map; and synthesizing the2D facial map into a 3D facial map.
 19. The image processing deviceaccording to claim 11, wherein the 2D facial map corresponds to a frontside angle of the 3D facial map.
 20. The image processing deviceaccording to claim 11, wherein the 3D model is a 3D human facial modelor a real person.